{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lb5yiH5h8x3h"
      },
      "source": [
        "##### Copyright 2025 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "906e07f6e562"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMGdicu8PVD9"
      },
      "source": [
        "# Video understanding with Gemini"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DR4Ti6Q0QKIl"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Video_understanding.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=30/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w14yjWnPVD-"
      },
      "source": [
        "Gemini has from the begining been a multimodal model, capable of analyzing all sorts of medias using its [long context window](https://developers.googleblog.com/en/new-features-for-the-gemini-api-and-google-ai-studio/).\n",
        "\n",
        "[Gemini models](https://ai.google.dev/gemini-api/docs/models/) bring video analysis to a whole new level as illustrated in [this video](https://www.youtube.com/watch?v=Mot-JEU26GQ):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CumMaR-sts53"
      },
      "outputs": [],
      "source": [
        "#@title Building with Gemini 2.0: Video understanding\n",
        "%%html\n",
        "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/Mot-JEU26GQ?si=pcb7-_MZTSi_1Zkw\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen></iframe>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jexx9acnuDsA"
      },
      "source": [
        "This notebook will show you how to easily use Gemini to perform the same kind of video analysis. Each of them has different prompts that you can select using the dropdown, also feel free to experiment with your own.\n",
        "\n",
        "You can also check the [live demo](https://aistudio.google.com/starter-apps/video) and try it on your own videos on [AI Studio](https://aistudio.google.com/starter-apps/video)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0HWzIEAQYqz"
      },
      "source": [
        "## Setup\n",
        "\n",
        "This section install the SDK, set it up using your [API key](../quickstarts/Authentication.ipynb), imports the relevant libs, downloads the sample videos and upload them to Gemini.\n",
        "\n",
        "Expand the section if you are curious, but you can also just run it (it should take a couple of minutes since there are large files) and go straight to the examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzBKAaL4QYq0"
      },
      "source": [
        "### Install SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "IbKkL5ksQYq1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/200.0 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m200.0/200.0 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "%pip install -U -q \"google-genai>=1.16.0\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDUGen_kQYq2"
      },
      "source": [
        "### Setup your API key\n",
        "\n",
        "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication ![image](https://storage.googleapis.com/generativeai-downloads/images/colab_icon16.png)](../quickstarts/Authentication.ipynb) for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "0H_lRdlrQYq3"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "\n",
        "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3Lez1vBQYq3"
      },
      "source": [
        "### Initialize SDK client\n",
        "\n",
        "With the new SDK you now only need to initialize a client with you API key (or OAuth if using [Vertex AI](https://cloud.google.com/vertex-ai)). The model is now set in each call."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "X3CAp9YrQYq4"
      },
      "outputs": [],
      "source": [
        "from google import genai\n",
        "from google.genai import types\n",
        "\n",
        "client = genai.Client(api_key=GOOGLE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITgsQyaXQYq4"
      },
      "source": [
        "### Select the Gemini model\n",
        "\n",
        "Video understanding works best with Gemini 2.5 models. You can also select former models to compare their behavior but it is recommended to use at least the 2.0 ones.\n",
        "\n",
        "For more information about all Gemini models, check the [documentation](https://ai.google.dev/gemini-api/docs/models/gemini) for extended information on each of them.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "IO7IoqbrQYq5"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.5-flash\" # @param [\"gemini-2.5-flash\", \"gemini-2.5-pro\",\"gemini-3-pro-preview\", \"gemini-2.5-flash-lite\", \"gemini-2.0-flash\"] {\"allow-input\":true, isTemplate: true}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rv8ULT0lvJ47"
      },
      "source": [
        "### Get sample videos\n",
        "\n",
        "You will start with uploaded videos, as it's a more common use-case, but you will also see later that you can also use Youtube videos."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "fMcwUw48vL1N"
      },
      "outputs": [],
      "source": [
        "# Load sample images\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/Pottery.mp4 -O Pottery.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/Jukin_Trailcam_Videounderstanding.mp4 -O Trailcam.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/post_its.mp4 -O Post_its.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/user_study.mp4 -O User_study.mp4 -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y4YMNQulz_yY"
      },
      "source": [
        "### Upload the videos\n",
        "\n",
        "Upload all the videos using the File API. You can find modre details about how to use it in the [Get Started](../quickstarts/Get_started.ipynb#scrollTo=KdUjkIQP-G_i) notebook.\n",
        "\n",
        "This can take a couple of minutes as the videos will need to be processed and tokenized."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "LUUMJ4kE0OZS"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/0w87yxp3d257\n",
            "Waiting for video to be processed.\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/qxfsdb4uy9jv\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/zbay420108wn\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/65xctq6o81eh\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "def upload_video(video_file_name):\n",
        "  video_file = client.files.upload(file=video_file_name)\n",
        "\n",
        "  while video_file.state == \"PROCESSING\":\n",
        "      print('Waiting for video to be processed.')\n",
        "      time.sleep(10)\n",
        "      video_file = client.files.get(name=video_file.name)\n",
        "\n",
        "  if video_file.state == \"FAILED\":\n",
        "    raise ValueError(video_file.state)\n",
        "  print(f'Video processing complete: ' + video_file.uri)\n",
        "\n",
        "  return video_file\n",
        "\n",
        "pottery_video = upload_video('Pottery.mp4')\n",
        "trailcam_video = upload_video('Trailcam.mp4')\n",
        "post_its_video = upload_video('Post_its.mp4')\n",
        "user_study_video = upload_video('User_study.mp4')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IF5tDbb-Q0oc"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "B0Z9QzC3Q2wX"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "from PIL import Image\n",
        "from IPython.display import display, Markdown, HTML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GAa7sCD7tuMW"
      },
      "source": [
        "# Search within videos\n",
        "\n",
        "First, try using the model to search within your videos and describe all the animal sightings in the trailcam video.\n",
        "\n",
        "<video controls width=\"500\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/Jukin_Trailcam_Videounderstanding.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "PZw41-lsKKMf"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "```json\n",
              "[\n",
              "  {\n",
              "    \"time\": \"00:00 - 00:17\",\n",
              "    \"caption\": \"Two gray foxes in the wild, foraging. One comes into view from the right, followed by another. They are sniffing the ground, and one climbs onto a rock.\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"00:17 - 00:34\",\n",
              "    \"caption\": \"A mountain lion is seen sniffing the ground in a forest, then briefly looking up and walking off. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"00:34 - 00:50\",\n",
              "    \"caption\": \"Two foxes are captured at night. One digs in the ground, and then they engage in a brief, aggressive interaction before running out of frame. (Night vision with IR flash)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"00:50 - 01:04\",\n",
              "    \"caption\": \"A bright flash occurs, followed by two foxes in a rocky area at night. They move around, one looks at the camera, and then another bright flash illuminates the scene. (Night vision with IR flash)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"01:04 - 01:17\",\n",
              "    \"caption\": \"A mountain lion walks from right to left across the frame in the dark. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"01:17 - 01:29\",\n",
              "    \"caption\": \"Two mountain lions are seen at night. The larger one walks past the camera in the foreground, while a smaller one (possibly a cub) walks on top of a rock in the background. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"01:29 - 01:51\",\n",
              "    \"caption\": \"A bobcat is seen at night, foraging on the ground, then digging a hole, and looking directly at the camera with glowing eyes. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"01:51 - 01:56\",\n",
              "    \"caption\": \"A brown bear walks away from the camera through a sun-dappled forest. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"01:56 - 02:04\",\n",
              "    \"caption\": \"A mountain lion walks into the frame from the left, looks at the camera, and then walks out of frame to the right. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"02:04 - 02:22\",\n",
              "    \"caption\": \"Two bears, possibly a mother and cub, are walking through the forest. One briefly obstructs the camera's view before they both move off into the distance. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"02:22 - 02:34\",\n",
              "    \"caption\": \"A fox is seen at night on a hill overlooking a city with twinkling lights. It sniffs the ground and then sits up to look out over the city. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"02:34 - 02:41\",\n",
              "    \"caption\": \"A bear walks past the camera at night, with a city lights landscape visible in the background. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"02:41 - 02:51\",\n",
              "    \"caption\": \"A mountain lion walks past the camera at night, with the illuminated city in the distance. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"02:51 - 03:04\",\n",
              "    \"caption\": \"A mountain lion walks towards a tree and then sniffs around on the ground. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"03:04 - 03:22\",\n",
              "    \"caption\": \"A brown bear stands in the forest, looks around, then directly at the camera, before walking off. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"03:22 - 03:40\",\n",
              "    \"caption\": \"Two brown bears are seen foraging on the ground in the forest. One bear briefly obstructs the camera's view as it moves closer. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"03:40 - 04:03\",\n",
              "    \"caption\": \"Two brown bears walk away from the camera. One sits down and scratches itself, then they both continue walking into the distance. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"04:03 - 04:22\",\n",
              "    \"caption\": \"Two brown bears walk towards the camera. One walks past, while the other remains in view, sniffing the ground. (Daylight)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"04:22 - 04:30\",\n",
              "    \"caption\": \"A bobcat with bright, glowing eyes looks at the camera, then walks past and out of frame. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"04:30 - 04:49\",\n",
              "    \"caption\": \"A fox appears in the distance with glowing eyes, walks closer to the camera, and then suddenly dashes out of frame. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"04:49 - 04:57\",\n",
              "    \"caption\": \"A fox is seen walking away from the camera into the dark forest. (Night vision)\"\n",
              "  },\n",
              "  {\n",
              "    \"time\": \"04:57 - 05:10\",\n",
              "    \"caption\": \"A mountain lion walks towards a tree, sniffs the ground, and then walks past the camera. (Night vision)\"\n",
              "  }\n",
              "]\n",
              "```"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"For each scene in this video, generate captions that describe the scene along with any spoken text placed in quotation marks. Place each caption into an object with the timecode of the caption in the video.\"  # @param [\"For each scene in this video, generate captions that describe the scene along with any spoken text placed in quotation marks. Place each caption into an object with the timecode of the caption in the video.\", \"Organize all scenes from this video in a table, along with timecode, a short description, a list of objects visible in the scene (with representative emojis) and an estimation of the level of excitement on a scale of 1 to 10\"] {\"allow-input\":true}\n",
        "\n",
        "video = trailcam_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        video,\n",
        "        prompt,\n",
        "    ]\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOQzKYGJKAnD"
      },
      "source": [
        "The prompt used is quite a generic one, but you can get even better results if you cutomize it to your needs (like asking specifically for foxes).\n",
        "\n",
        "The [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows how you can postprocess this output to jump directly to the the specific part of the video by clicking on the timecodes. If you are interested, you can check the [code of that demo on Github](https://github.com/google-gemini/starter-applets/tree/main/video)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wog32E7CKnT6"
      },
      "source": [
        "# Extract and organize text\n",
        "\n",
        "Gemini models can also read what's in the video and extract it in an organized way. You can even use Gemini reasoning capabilities to generate new ideas for you.\n",
        "\n",
        "<video controls width=\"400\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/post_its.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "baNCeA3GKrfu"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Here are the transcribed project names from the sticky notes, organized alphabetically in a table, along with a few more ideas:\n",
              "\n",
              "## Brainstorm: Project Names\n",
              "\n",
              "| Project Name         | Project Name         |\n",
              "| :------------------- | :------------------- |\n",
              "| Aether               | Leo Minor            |\n",
              "| Andromeda's Reach    | Lunar Eclipse        |\n",
              "| Astral Forge         | Lyra                 |\n",
              "| Athena               | Lynx                 |\n",
              "| Athena's Eye         | Medusa               |\n",
              "| Bayes Theorem        | Odin                 |\n",
              "| Canis Major          | Orion's Belt         |\n",
              "| Celestial Drift      | Orion's Sword        |\n",
              "| Centaurus            | Pandora's Box        |\n",
              "| Cerberus             | Persius Shield       |\n",
              "| Chaos Field          | Phoenix              |\n",
              "| Chaos Theory         | Prometheus Rising    |\n",
              "| Chimera Dream        | Riemann's Hypothesis |\n",
              "| Comets Tail          | Sagitta              |\n",
              "| Convergence          | Serpens              |\n",
              "| Delphinus            | Stellar Nexus        |\n",
              "| Draco                | Stokes Theorem       |\n",
              "| Echo                 | Supernova Echo       |\n",
              "| Equilibrium          | Symmetry             |\n",
              "| Euler's Path         | Taylor Series        |\n",
              "| Fractal              | Titan                |\n",
              "| Galactic Core        | Vector               |\n",
              "| Golden Ratio         | Zephyr               |\n",
              "| Hera                 |                      |\n",
              "| Infinity Loop        |                      |\n",
              "\n",
              "---\n",
              "\n",
              "## A Few More Project Name Ideas:\n",
              "\n",
              "1.  **Pulsar:** (Astronomical, suggests powerful and rhythmic energy)\n",
              "2.  **Axiom:** (Mathematical/logical, implies a fundamental truth or starting point)\n",
              "3.  **Artemis:** (Mythological, associated with precision, exploration, and the moon)\n",
              "4.  **Quantum Leap:** (Scientific, indicates a significant and sudden advancement)\n",
              "5.  **Vortex:** (Implies a central point of activity, energy, or convergence)"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Transcribe the sticky notes, organize them and put it in a table. Can you come up with a few more ideas?\" # @param [\"Transcribe the sticky notes, organize them and put it in a table. Can you come up with a few more ideas?\", \"Which of those names who fit an AI product that can resolve complex questions using its thinking abilities?\"] {\"allow-input\":true}\n",
        "\n",
        "video = post_its_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        video,\n",
        "        prompt,\n",
        "    ]\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjKIsLDMTNk1"
      },
      "source": [
        "# Structure information\n",
        "\n",
        "Gemini is not only able to read text but also to reason and structure about real world objects. Like in this video about a display of ceramics with handwritten prices and notes.\n",
        "\n",
        "<video controls width=\"500\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/Pottery.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "bqzqedMFT5Wp"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Here's a table summarizing the items and notes from the image:\n",
              "\n",
              "| Category          | Item                | Description                                                                                                                                                                                                                             | Dimensions                     | Price   | Additional Notes                  |\n",
              "| :---------------- | :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------- | :------ | :-------------------------------- |\n",
              "| Drinkware         | Tumblers            | Stacked and individual tumblers with an earthy brown/beige base and a light blue/white wavy glaze towards the top. Two small, round ceramic samples displaying the base and blue/grey glaze are shown next to them. | 4\"h x 3\"d (approx.)            | \\$20    | \\#5 Artichoke double dip          |\n",
              "| Bowls             | Small Bowls         | Two bowls with a speckled, rustic brown/orange exterior and a darker, possibly iridescent, interior with hints of blue/green.                                                                                                        | 3.5\"h x 6.5\"d                  | \\$35    |                                   |\n",
              "| Bowls             | Medium Bowls        | Two larger bowls, similar in appearance to the small bowls with a speckled, rustic brown/orange exterior and a darker, iridescent interior with hints of blue/green.                                                               | 4\"h x 7\"d                      | \\$40    |                                   |\n",
              "| Glaze Sample/Test | Gemini Double Dip   | A rectangular ceramic tile with \"6b6\" inscribed, displaying a brown/rust speckled glaze on one side and a blue/grey glaze on the other.                                                                                              | N/A (sample tile)              | N/A     | \\#6 Gemini double dip, Slow Cool |"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Give me a table of my items and notes\" # @param [\"Give me a table of my items and notes\", \"Help me come up with a selling pitch for my potteries\"] {\"allow-input\":true}\n",
        "\n",
        "video = pottery_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        video,\n",
        "        prompt,\n",
        "    ],\n",
        "    config = types.GenerateContentConfig(\n",
        "        system_instruction=\"Don't forget to escape the dollar signs\",\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zsh6i-Z6VHNK"
      },
      "source": [
        "As you can see, Gemini is able to grasp to with item corresponds each note, including the last one."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uIfsFC0pVUTD"
      },
      "source": [
        "# Analyze screen recordings for key moments\n",
        "\n",
        "You can also use the model to analyze screen recordings. Let's say you're doing user studies on how people use your product, so you end up with lots of screen recordings, like this one, that you have to manually comb through.\n",
        "With just one prompt, the model can describe all the actions in your video.\n",
        "\n",
        "<video controls width=\"400\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/user_study.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "wrMHZ0MxW75y"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "The \"My Garden App\" is presented, showcasing a list of various plants with their prices, alongside \"Like\" and \"Add to Cart\" buttons. (0:00-0:09) The user demonstrates interacting with the app by liking several plants, which changes the \"Like\" button to red, and adding multiple items to the shopping cart, confirmed by an \"Added!\" message. (0:09-0:25, 0:41-0:45)\n",
              "After adding items, the user navigates to the \"Cart\" tab to view the selected plants and their total cost. (0:30-0:33) Finally, the \"Profile\" tab provides a summary of the user's activity, displaying the number of liked plants and items in the cart. (0:33-0:35)"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Generate a paragraph that summarizes this video. Keep it to 3 to 5 sentences with corresponding timecodes.\" # @param [\"Generate a paragraph that summarizes this video. Keep it to 3 to 5 sentences with corresponding timecodes.\", \"Choose 5 key shots from this video and put them in a table with the timecode, text description of 10 words or less, and a list of objects visible in the scene (with representative emojis).\", \"Generate bullet points for the video. Place each bullet point into an object with the timecode of the bullet point in the video.\"] {\"allow-input\":true}\n",
        "\n",
        "video = user_study_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        video,\n",
        "        prompt,\n",
        "    ]\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TEYYemjyKcZ7"
      },
      "source": [
        "# Analyze youtube videos\n",
        "\n",
        "On top of using your own videos you can also ask Gemini to get a video from Youtube and analyze it. He's an example using the keynote from Google IO 2023. Guess what the main theme was?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "DP0Dd0hJKvYm"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Sundar says \"AI\" at the following instances:\n",
              "\n",
              "1.  **0:29 - 0:32**: \"As you may have heard, AI is having a very busy year.\"\n",
              "    *   **Broader context**: Sundar is beginning his keynote address, noting the significant activity in the field of AI, indicating it will be a major topic of discussion.\n",
              "2.  **0:39 - 0:40**: \"Seven years into our journey as an AI-first company.\"\n",
              "    *   **Broader context**: He emphasizes Google's long-standing commitment to AI, framing their current advancements within this established focus.\n",
              "3.  **0:46 - 0:47**: \"We have an opportunity to make AI even more helpful...\"\n",
              "    *   **Broader context**: He highlights Google's vision for AI to be universally beneficial for individuals, businesses, and communities.\n",
              "4.  **0:54 - 0:56**: \"We've been applying AI to make our products radically more helpful for a while.\"\n",
              "    *   **Broader context**: He states that AI has already been integrated into Google's products to enhance their utility.\n",
              "5.  **0:59 - 1:00**: \"With generative AI, we are taking the next step.\"\n",
              "    *   **Broader context**: He introduces generative AI as the next phase of development for Google's products, starting with Gmail.\n",
              "6.  **1:16 - 1:19**: \"Let me start with few examples of how generative AI is helping to evolve our products, starting with Gmail.\"\n",
              "    *   **Broader context**: He transitions to practical examples of how generative AI is being used in Google's core products.\n",
              "7.  **1:40 - 1:42**: \"Smart Compose led to more advanced writing features powered by AI.\"\n",
              "    *   **Broader context**: Discussing the evolution of Gmail's smart features, he attributes their advanced capabilities to AI.\n",
              "8.  **3:03 - 3:05**: \"Since the early days of Street View, AI has stitched together billions of panoramic images...\"\n",
              "    *   **Broader context**: He explains how AI has been instrumental in creating immersive experiences in Google Maps' Street View.\n",
              "9.  **3:14 - 3:16**: \"Immersive View, which uses AI to create a high fidelity representation of a place...\"\n",
              "    *   **Broader context**: He describes Immersive View's technology, crediting AI for generating realistic 3D representations.\n",
              "10. **5:15 - 5:17**: \"It was one of our first AI-native products.\"\n",
              "    *   **Broader context**: Referring to Google Photos, he emphasizes its foundational reliance on AI since its inception.\n",
              "11. **5:30 - 5:32**: \"We also want to help you make them better. In fact, every month, 1.7 billion images are edited in Google Photos.\"\n",
              "    *   **Broader context**: Sundar explains how Google Photos leverages AI to enable advanced photo editing features like Magic Eraser.\n",
              "12. **5:41 - 5:43**: \"AI advancements give us more powerful ways to do this.\"\n",
              "    *   **Broader context**: He highlights that recent AI breakthroughs are enhancing photo editing capabilities within Google Photos.\n",
              "13. **5:48 - 5:50**: \"Magic Eraser, launched first on Pixel, uses AI-powered computational photography to remove unwanted distractions.\"\n",
              "    *   **Broader context**: He specifies Magic Eraser's use of AI for computational photography.\n",
              "14. **7:40 - 7:44**: \"These are just a few examples of how AI can help you in moments that matter.\"\n",
              "    *   **Broader context**: He summarizes the various product examples (Gmail, Photos, Maps) as demonstrations of AI's helpfulness.\n",
              "15. **7:47 - 7:49**: \"And there is so much more we can do to deliver the full potential of AI...\"\n",
              "    *   **Broader context**: He expresses optimism about the future potential of AI across Google's product ecosystem.\n",
              "16. **8:24 - 8:26**: \"Making AI helpful for everyone is the most profound way we will advance our mission.\"\n",
              "    *   **Broader context**: He articulates Google's overarching mission: to make AI accessible and beneficial to all.\n",
              "17. **8:31 - 8:33**: \"And we are doing this in four important ways. First, by improving your knowledge and learning...\"\n",
              "    *   **Broader context**: He begins to outline Google's four key strategies for deploying AI helpfully and responsibly.\n",
              "18. **8:53 - 8:57**: \"And finally, by building and deploying AI responsibly so that everyone can benefit equally.\"\n",
              "    *   **Broader context**: He concludes the four strategic pillars with the emphasis on responsible AI development and deployment.\n",
              "19. **9:03 - 9:08**: \"Our ability to make AI helpful for everyone relies on continuously advancing our foundation models.\"\n",
              "    *   **Broader context**: He reiterates the goal of making AI helpful for everyone, linking it directly to the development of foundation models.\n",
              "20. **11:26 - 11:31**: \"It uses AI to better detect malicious scripts and can help security experts understand and resolve threats.\"\n",
              "    *   **Broader context**: He discusses Sec-PaLM, an AI model fine-tuned for security, demonstrating AI's application in cybersecurity.\n",
              "21. **13:00 - 13:02**: \"These teams have contributed to a significant number of them: AlphaGo, Transformers, word2vec, WaveNet, AlphaFold, Sequence to sequence models, Distillation, Deep reinforcement learning.\"\n",
              "    *   **Broader context**: Sundar reviews Google's historical contributions to AI breakthroughs, highlighting the achievements of their Brain and DeepMind teams.\n",
              "22. **13:10 - 13:13**: \"All this helps set the stage for the inflection point we are at today.\"\n",
              "    *   **Broader context**: He concludes a segment on Google's AI foundational models and their impact, setting the stage for future developments.\n",
              "23. **13:25 - 13:28**: \"They are focused on building more capable systems safely and responsibly.\"\n",
              "    *   **Broader context**: Discussing the unified Google DeepMind team, he emphasizes their commitment to building advanced AI systems with safety and responsibility.\n",
              "24. **14:07 - 14:11**: \"As we invest in more advanced models, we are also deeply investing in AI responsibility.\"\n",
              "    *   **Broader context**: He highlights Google's commitment to responsible AI development alongside advancements in model capabilities.\n",
              "25. **15:10 - 15:12**: \"James will talk about our responsible approach to AI later.\"\n",
              "    *   **Broader context**: He introduces an upcoming segment dedicated to discussing Google's responsible AI strategies.\n",
              "26. **15:28 - 15:30**: \"That's the opportunity we have with Bard, our experiment for conversational AI.\"\n",
              "    *   **Broader context**: He introduces Bard as Google's experimental conversational AI."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=types.Content(\n",
        "        parts=[\n",
        "            types.Part(text=\"Find all the instances where Sundar says \\\"AI\\\". Provide timestamps and broader context for each instance.\"),\n",
        "            types.Part(\n",
        "                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=ixRanV-rdAQ')\n",
        "            )\n",
        "        ]\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zN0296794Lui"
      },
      "source": [
        "# Customizing video preprocessing\n",
        "\n",
        "The Gemini API allows you to define some preprocessing steps to enhance your abilities to understand and extract information from videos.\n",
        "\n",
        "You can use clipping intervals (or define time offsets to focus on specific video parts) and custom FPS (to define how many frames will be considered to analyze the video.\n",
        "\n",
        "For more details about those features, you can take a look at the [Customizing video preprocessing](https://ai.google.dev/gemini-api/docs/video-understanding#customize-video-preprocessing) at the Gemini API documentation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LEUu4qCY-V7r"
      },
      "source": [
        "## Analyze specific parts of videos using clipping intervals\n",
        "\n",
        "Sometimes you want to look for specific parts of your videos. You can define time offsets on your request, pointing to the model which specific video interval you are more interested about.\n",
        "\n",
        "**Note:** The `video_metadata` that you will inform must be representing the time offsets in seconds.\n",
        "\n",
        "In this example, you are using this video, from [Google I/O 2025 keynote](https://www.youtube.com/watch?v=XEzRZ35urlk) and asking the model to consider specifically the time offset between 20min50s and 26min10s."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "qeLcdeu95Bd9"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Here is a 3 sentence summary of the video:\n",
              "\n",
              "Demis Hassabis introduces Google DeepMind's newest developments in AI, focusing on improving AI’s understanding and interaction with the world. He highlights the launch of Gemini 1.5 Flash, a multimodal model for efficiency and fast service, alongside the announcement of Project Astra, aimed to develop a universal AI agent helpful in everyday life. The talk emphasizes enhancing natural, conversational interaction and personalized responsiveness in AI assistants."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=types.Content(\n",
        "        parts=[\n",
        "            types.Part(\n",
        "                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=XEzRZ35urlk'),\n",
        "                video_metadata=types.VideoMetadata(\n",
        "                    start_offset='1250s',\n",
        "                    end_offset='1570s'\n",
        "                )\n",
        "            ),\n",
        "            types.Part(text='Please summarize the video in 3 sentences.')\n",
        "        ]\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7JL2UeuT6X7k"
      },
      "source": [
        "You can also use clipping intervals for videos uploaded to the File API as also inline videos on your prompts (remembering that inline data cannot exceed 20MB in size)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "mluu__Cw6ktt"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Here are the key events from the trail camera footage:\n",
              "\n",
              "* **0:00-0:16:** Two grey foxes move around a rocky area.\n",
              "* **0:17-0:34:** A mountain lion explores a wooded area.\n",
              "* **0:35-0:50:** Two foxes play/fight, one gets tossed into the air.\n",
              "* **0:51-1:16:** Two mountain lions, likely a mother and cub, move through a rocky area at night.\n",
              "* **1:17-1:28:** Two mountain lions walk toward the camera.\n",
              "* **1:29-1:50:** A bobcat explores a wooded area at night.\n",
              "* **1:51-2:22:** Two young bears walk toward and investigate the trail camera.\n",
              "* **2:23-2:51:** A fox and then a bear pass by a scenic overlook with a city lit up in the distance.\n",
              "* **2:52-3:04:** A mountain lion approaches and scratches at something in the ground.\n",
              "* **3:05-3:21:** A bear walks toward the trail camera and starts panting.\n",
              "* **3:22-4:19:** A bear followed by a bear cub are seen walking in the woods, followed by a mountain lion approaching the camera. \n",
              "* **4:22-4:56:** A bobcat walks along a log and looks at the trail camera.\n",
              "* **4:57-5:09:** A mountain lion explores the area, smelling the ground."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Summarize this video in few short bullets\"  # @param [\"For each scene in this video, generate captions that describe the scene along with any spoken text placed in quotation marks. Place each caption into an object with the timecode of the caption in the video.\", \"Organize all scenes from this video in a table, along with timecode, a short description, a list of objects visible in the scene (with representative emojis) and an estimation of the level of excitement on a scale of 1 to 10\"] {\"allow-input\":true}\n",
        "\n",
        "video = trailcam_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=types.Content(\n",
        "        parts=[\n",
        "            types.Part(\n",
        "                file_data=types.FileData(\n",
        "                    file_uri=video.uri,\n",
        "                    mimeType=video.mime_type),\n",
        "                video_metadata=types.VideoMetadata(\n",
        "                    start_offset='60s',\n",
        "                    end_offset='120s'\n",
        "                )\n",
        "            ),\n",
        "            types.Part(text=prompt)\n",
        "        ]\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZGz46VP8pE5"
      },
      "source": [
        "## Customize the number of video frames per second (FPS) analyzed\n",
        "\n",
        "By default, the Gemini API extract 1 (one) FPS to analyze your videos. But this amount may be too much (for videos with less activities, like a lecture) or to preserve more detail in fast-changing visuals, a higher FPS should be selected.\n",
        "\n",
        "In this scenario, you are using one specific interval of one Nascar pit-stop as also you will capture a higher number of FPS (in this case, 24 FPS)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "_6OhwEH0-zHq"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "According to the video, only the tires on the left side of the car were replaced. "
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=types.Content(\n",
        "        parts=[\n",
        "            types.Part(\n",
        "                file_data=types.FileData(file_uri='https://www.youtube.com/watch?v=McN0-DpyHzE'),\n",
        "                video_metadata=types.VideoMetadata(\n",
        "                    start_offset='15s',\n",
        "                    end_offset='35s',\n",
        "                    fps=24\n",
        "                )\n",
        "            ),\n",
        "            types.Part(text='How many tires where changed? Front tires or rear tires?')\n",
        "        ]\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cizoUEdIYLd0"
      },
      "source": [
        "Once again, you can check the  [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows an example on how to postprocess this output. Check the [code of that demo](https://github.com/google-gemini/starter-applets/tree/main/video) for more details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lND4jB6MrsSk"
      },
      "source": [
        "# Next Steps\n",
        "\n",
        "Try with you own videos using the [AI Studio's live demo](https://aistudio.google.com/starter-apps/video) or play with the examples from this notebook (in case you haven't seen, there are other prompts you can try in the dropdowns).\n",
        "\n",
        "For more examples of the Gemini capabilities, check the other guide from the [Cookbook](https://github.com/google-gemini/cookbook/). You'll learn how to use the [Live API](../quickstarts/Get_started_LiveAPI.ipynb), juggle with [multiple tools](../quickstarts/Get_started_LiveAPI_tools.ipynb) or use Gemini's [spatial understanding](../quickstarts/Spatial_understanding.ipynb) abilities.\n",
        "\n",
        "The [examples](https://github.com/google-gemini/cookbook/tree/main/examples/) folder from the cookbook is also full of nice code samples illustrating creative ways to use Gemini multimodal capabilities and long-context."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Video_understanding.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
